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ABSTRACT 

Evaluation roles used in the Department of Research 
and Development in the Cincinnati Public Schools are identified and 
described^ These include: project evaluator, local**school evaluator, 
independent program evaluator, external evaluator and external 
auditors. The merits of each evaluation role will be discussed as to 
its relationship with credibility^ objectivity, independence and 
usefulness. The basis for judging the merits of each evaluation role 
vith regard to the above four criteria vill be: (1) types of 
decisions to be made; and (2) safeguards to maximize each of the four 
criteria. (Autkor/RC) 
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VoJ Abstract 

This paper will identify and describe the evaluation roles usad In 
the Department of Research and Development in tho Cincinnati Public Schools. 
These include; pro,iect evaluator, local«*school evaluator^ independent program 
evaluator^ external evaluator and external auditors. 

The merits of each evaluation role will be disc*issed as to its relations- 
ship with credibility 9 objectivity^ Independence and usefulness. The basis 
for judging the merits of each evaluation role with regard to the above 
four criteria will be (1) typej^ of decisions to be made and (2) safeguards 
to maximize each of the four (Criteria. 
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SPECTRUM OF OBJECTIVITY - CREDIBILITY IN EVALUATION 



Uahal R. Ahn^ Maryaon M. Bresnan* Joseph F. Gastright and Gerald Varland^ 
ClndLnaatl Public Schools* 

Purpose 

This paper will identify and describe the evaluation roles used in 
the Department of Research and Development in the Cincinnati Fublic Schools. 
These include: project evaluator, local«*8Chool evaluator, independent program 
evaluator^ external evaluator and external auditor. The report will discuss 
methods used to assure an appropriate level of objectivity to the findings. 

Background 

The i&sue of objectivity In evaluation is so central to the process 
itself that many discussions of it are shrouded in defensive language. The 
final act of placing value » whether performed by the decision-maker or by 
the evaluator 9 extends beyond the fringes of method into the realm of decisions- 
making itself. The denial of this subjective element-- or at least the attempt 
to bury it in jargonr-is a major reason for much of the turgici prose about 
evaluation. The scientific method contains subjective elements of the same- 
sort (Kaplant 1964) ^ which are often buried in the research literature. 

The "evaluation as researdi** proponents (Suchman^. 1967} offer objectivity 

* ' 1 

in the form of experimental controls and appropriate data analysis. Others 
suggest that the presence of a sound basis in theory is a desirable characteristic 
of evaluative research (Bernstein^ 1973} « Tliese ''scientific*' procedures act 
te reduce bias^ hut do not guarantee that the findings are either useful or 
even "true** (House ^ 1972). 
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Eellance on research methodology has been identified as a major 
cause of irrelevant evaluation reports (Stuff lebeam ettal*^ 1971} • 

The '^evaluation as information** propoiients (Stufflebeam et«al«9 1971} 
accept ''usefulness" and presumably "true" as the primary concern of 
evaluation* Pre--ordinate concern of utility evaluation is the creation 
maintenance of credibility by the evaluator* 

Opposing sets of values held by various "stakeholders" in a project can 
effect the credibility of "informative evaluations" which pass through many 
hands (Johnson^ 1971}* The credibility of internal evaluations can be 
increased by in external avdtt of the results (Kniefel, 1971)* In some 
cases internal "Independent" evaluators are suf^gested as a balance between 
objectivity and usefulness (P^ay and Costello* 1974} • This balance can be 
affected by the type of decision being made and the evaluation budget* 

Assumptions and Definitions 

Given a choice* every evaluator would prefer to produce obji^ctlve* 
independent^ credible and useful evaluations* However* it is an assumption 
of this paper that the safeguards an institution can apply in practice can- 
not maximize all the above characteristics at the same time* 

Independence is defined as the autonomy of fmction the evaluator is 
given within organizational contexts* It is a guarantee that evaluation in 
design* direction* and interpretation is free from interference* subjection 
and influence* The evaluator *8 independence requires exemption from both 
control und support* In other words » an extremely independent evaluation 
may be similar to an adversary evaluation* The goal free evaluation 
described by Scriven approaches this independent ideal* 

In organizational contexts* independence involves the explication of 
those aseuBptlons and a reporting relationship which insures freedom from 
O subjection and influence* These factors* as part of the evaluator 's role 



definition^ delimit the guaranteed iDdependence of an evaluator from 

the viewpoint of the organization within which he operates* The characteT.'istics 

of independence as it applies to the evaluator*s role in a definition may 

not be the same in actual practice* When conflict arises t> the evaluator 

selects the alternative which is more reality based* Personal independence ^ 

expertise and experience » while valued in theory » do not always safeguard the 

independence of evaluation activities* 

Credibility represents the quality of trust that is established between 
the evaluator and the user of evaluative Information* Beyond the scientific 
criteria of evaluation data^ the user needs to be able to trust the 
evaluator in order to make use of the data* For this reason it is an assui^>tion 
of this paper that credibility can be maximized only by a high degree of 
mutual trust between the evaluator and cue specific management level for 
which the evaluation results are target^id* 

In organizational terms » credibility can be safeguarded by requiring 
pre-*specification of mutually agreeable outcomes by the appropriate manage- 
ment level and by allotting sufficient time and effort to develop mutual 
understanding* The more commensurate the outcomes and under£ttand:*.ng8 are 
within some methodolical viewpoint^ the more likely the results will be 
thought of as credible* 

As you will note 9 independence is difficult to maximize in conjunction 
with credibility* If the role and viewpoints of the evaluator are in conflict 
with the viewpoints and assumptions of the user or a management personnel^ 
thetis conflicts need to be solved through deliberate efforts to establish 
a working relationship between the two* However ^ the very effort can easily 
destroy the Independence of the evaluator* 



Objectivity Is defined In this paper as validity and reliability of 
evaluation Information* It Is our contention that this has been a traditional 
criteria for Judging the soundness of research and evaluation* The attempt 
to guarantee objectivity in evaluation is not only a design problem but also 
selecting the best design which can fit the program in the field* More often 
the evaluator is ccmfronted with the selection of the most objective 
nethodology which can be applied, given the operational details of the treat* 
sent* 

Objectivity is safeguarded by presfecification of those aspects of 
behavior which will be measured* Ihls Includes a systematic procedure for 
observation* The objectivity of evaluation results will then be limited by 
the methodology chosen for a particular evaluation* 

Usefulness represents the degree to which evaluation results are used* 
Since the user cf evaluation lnform&tlon is most often someone other than 
the evaluator, it is difficult for the evaluator to Insure usefulness of 
data* The evaluation Infcrmatlon must address Itself to a specific manage* 
ment level which will use the infomatlon* 

Usefulness can be safeguarded by insuring that evaluators and management 
personnel have the time to arrive at agreement about the outcomes to be used* 
Usefulness requires that project decision making and project operation build 
a mechanism for evaluation feedback and subsequent modifications* Evaluation 
must be fleadble enough tc answer meaningful questions which arise at 
different stages of a projects Ul^ly credible, and objective answers to the 
wrong questions are not useful* Ul^ly useful data sometimes are non Mghly 
objective* 

The safeguards which Insure independence and credibility are much 
concerned with the relationship between the evaluator and the user, and the 
role definition of the evaluator within an organization* The safeguards 



which Insure objectivity and usefulness are concerned with the quality of 
data provided and the responsiveness of the 4ata to the particular project's 
needs* 

Thus far^ ve have defined the four crlterla-^lndependence^ credibility^ 
objectivity 9 and usefulne8S~and explained the underlying assumptions* Our 
assuaq>tion is^ again^ that these criteria cannot be safeguarded within a 
single evaluator's role nor can they be maximized at the same time* 

For the purpose of this paper » we have developed a matrix showing the 
evaluator roles on one side and the four criteria on the other side* Then 
each role is rated as high^ medium or low in terms of independence^ 
credibility 9 objectivity » and usefulness* 

It should be reminded that independence and credibility criteria are 
related more to the evaluator*s relationship with the user^ while objectivity 
and usefulness concern more with the quality of data provided* The ratings 
of each role are presented in Figure 1, 



Figure 1. Ratings of Independence, Credibility, Objectivity and Usefulness 
for Five Evaluator Roles* 
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Criteria Evaluator Related Data Related 

SSlSS. Independence CredihiUty Objectivity Usefulness 

^^oiect low MBdiun Madlum Medium 
^ "^8^ 

Local School High Madiuiu High High 

Evaluator Medium High Medium 

Independent High Hadlum High Madium 
Program Evaluator 

Hi«»» Medium Hish Medium 

Evaluator 

High Medium High Medium 



The ratings do not represent the relative Importance placed on the 
criteria, but rather the characteristics that have been safeguarded. Those 
safeguards are more or less built Into the evaluator's role which may be 
commonly found In a large school system* 

In the discussion section ve will define each of the evaluator roles 
vhl'zh have been used In Cincinnati Public Schools. The definition of roles 
Includes major decision types the evaluator deals with* The decision types 
axe the\klnds of questions the evaluator must answer In order to fulfill 
Informatron requirements during different stages of program development and 
operation* 

The types of questions the evaluator may deal with Include the following: 
!• What is the problem^needs assessment? 
2. ' What strategy is approprlate~lnput evalL^tion? 
3« Is the project operating as planned«^process evaluation? 

4. Did it meet its obJectlves-*->output evaluation? 

5. Should we continue or discontinue the project-^ Impact evalu^.tlon? 
The discussion which follows next will explain this further for each 

evaluator role and describe how the ratings of criteria illuminate meanings 

into operational contents • 

Discussion 

The discussion section of the paper includes the interactioa of the 
varicus evaluator's roles » the major emphasis with regard to objectivity^ 
Independence 9 credibility and usefulness and hew these are altered by the 
stage of the project. It also includes nhe safeguards the system has built 
into each role with regard to objectivity » independence^ credibility and 
usefulness. 

Project Evaluator > The project evaluator in the Cincinnati Public 
Schools conducts evaluation of developtaental and innovative projects 
(basically Title III, E.StEaA.) and reports directly to the project coordinator. 



The roles this person plays at various stages of the project are varied. 



During forxpulatlon of the project^ the principal role of an evaluator 
Is to help the program planners to approriately define their objectives , and 
to forwilate criteria for the asaeaamect of the objectives* However ^ the 
evaluator who does this is usually not the person who ends up being the 
project evaluator^ since ve are applying for a grant during the formulation 
stage* 

The evaluator should be instrumental in having evaluation built into 
the program design so that the evaluation requirements are a recognized and 
accepted aspect of the program operuion^ ^^ther than sup .rimposed and in 
conflict vita other operations* Again, it should-be noted that ^he project 
evaluator has usually not been involved during these stages and must accept , 
for the most part, what was written into the grant application* 

During implementation and operation of the project, the evaluator serves 
to provide process and product information to the project staff, on the basis 
of which the project may be modified* This function constitutes serving as a 
diange agent or connective feedback loop in the development of the project* 
Purposes and objectives may be refinea and modified as a result of the 
feedback, as well as features of the project operation* 

We use a project evaluator in our Title III programs because we feel that 
as a meaner of the project staff, this person will be regarded as knowing 
and appreciating the nature of the project, as being accepted by the reat of 
the staff and as being able to operate with little disruptive influence* Since 
these projects are developmental in nature, we feel that it is a necessity that 
such a position be available within each project* However, we do note that by 
virtue of this person's intimate involvement with the program, he/she may be 
uasble to interpret findings in an appropriate perspective, thereby limiting 
the objectivity of the evaluation* 
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However I If a good working relationship does exist within the project ^ 
the evaluator's credibility and the usefulness of the data will be high* 
Therefore^ the independence factor is low with regard to the project evaluator* 
We have chosen to accept the loss of independence since the credibility factor 
will increase with the evaluator serving as part of a team* The project 
evaluator thus becomes a staff meober reporting to the project coordinator 
with an expertise in evaluation as opposed to a staff member with expertise 
in media* 

Post-project functions of the evaluator include the comparison of end- 
products with the stated goals to assess the degree of success and then 
furnish information to decision-makers as to what might be expected should the 
project be continued^ expanded or replicated* 

Here 9 the usefulness of data will be very high since the project evaluator 
has been intimately involved with the project* Also^ the objectivity of the 
data should be high since many of the objectives were already set up before 
the project evaluator was employed and it was up to the evaluator i;o collect 
the data to measure their pre-determined objectives* However due to the intimate 
involvement^ the objectivity may be somewhat compromised* 

In most cases I decisions of a go-no-go nature are based on the financial 
situation of the school board* Yet» among those programs competing for 
funds I those which have more defensible results will be considered in a 
better light* Here^ the project evaluator is seen as an ^^expert'' in evaluation 
and is considered to be independent from the mainstream of activities within 
the regular school curriculum* 

In projects^ many times the objectives are project-based and do not give 
the impact-type and generalizability of data which people sometimes need for 
go-no-go decisions* - 
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Local School Evaluator 

The local school evaluator (LSE) in the Clncixmati Public Schools 
provides evaluation services to local schools and works directly with the 
principal and the staff of a scliool* Since a school program is broader in 
scope and more routinized in activities than a single project » the type of 
questions the local school evaluator deals with are very diverse* 

During the stage when a problem is not clearly defined » the role of 
the local school evaluator is to help a school staff identify the area of 
need for that particular school* A starting point is usually tlie examination 
of existing data within the school system* This includes test data» survey 
data I and data such as attendance and mobility from a computerized central file* 
This enables a school to verify their perceived needs and quite often identify 
new areas of need* The local school evaluator* s major eo^hasis is on this 
stage* During this stage the local school evaluator can provide a perspective 
which may be more objective and data^based than the sdiool personnel* The 
local school evaluator's role at this time maximizes the objectivity of the 
data and the independence of the evaluator* 

In later stages there will be aderemphasis on independence and objectivity 
and more eiqphasis on credibility and usefulness* It should be noted that each 
school is unique and» therefore » the kinds of services are tailored to the 
needs of each school* 

In other stages^ the data provided by the local school evaluator are 
selected primarily in terms of their usefulness* The data provided are 
objectivei but the way they are selected may be based on subjective judgment* 
Therefore I the data provided by the local school evaluator are rated high 
in usefulness and rated medium in objectivity* 

During the design phase » when a specific instructional treatment is 
developed » the local school evaluator is not directly involved* This kind 

ERLC 



10 

of tnpxxt has been sought traditionally from the instructional consultant. 
A local school evaluator can function in this stage by helping the classroom 
teacher formulate behavioral objectives. This may be a logical extension 
of needs assessment » and it is essential in designing instructional plans. 
Another example of the local school evaluator activity may be presenting infor- 
mation about different types of criterion- referenced tests and their usages for 
classroom instruction. 

Data provided at this stage are highly useful, since the local school 
evaluator responds to the unique needs of the school. The information 
is usually very objective. Since the local school evaluator is independent from 
the major operations of a school program, he/she is moderately credible in 
providing input evaluation to a local school. 

When addressing the question, "Is it operating as planned?", the local 
school evaluator strives to provide the program implementors with useful 
infoimation. This is possible because the evaluator is monitoring the iiq>ortant 
process activities (rated by school staff). Although the data provided are 
objective, the selection of the dimensions to be monitored may be subjective. 
Independence at this stage is de-emphasized. Monitoring the implementation 
of a program is an activity of the local school evaluator. However, quite 
often the school system does not have the additional resources to respond to 
such demands. The demand for this service is not so great at this point. A 
aimllar constraint limits the role of the local school evaluator in dealing 
with evaluation of end results. 

The local school evalautor is not directly Involved with providing data 
for go-no-go dedsf-^n; however, the information used for the preceding stages 
can be use':! for this purpose. 

Usefulness of findings and credibility of the evaluator are maximized by 
the role of the local school evaluator, although independence and objectivity 
ERJC my not be maidmized to the same degree. 

* • • 12 
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Independent Program Evaluator t The independent program evaluator in 
the Cincinnati Public Schools conducts evaluation of federally funded or 
8tate funded programs. These programs are always admini.8tered from another 
department and quite often the coordinators are school based. Although this 
person performs many functions during the course of the project » the main 
emphasis is product evaluations. 

The independent program evaluator usually is not involved during the 
initial stages of project development. Although assistance is given by a 
person with evaluation skills i this person is not necessarily the independent 
program evaluator that will continue working on the project once the program 
is accepted by the funding agency. 

Once the independent project fd valuator is assigned to a particular project » 
he/she checks the evaluation design and modifies it when necessary* The main 
emphasis is on prespecified process and product objectives. 

Independence in both cases is built into this role by physical location 
of indep<^adent program evaluator and his/her direct reporting relationships. 
The independent project evaluator reports to the coordinator of Program 
Evaluation rather than directly to the project coordinator. 

The intent of providing projects with process and product information is 
'to give them feedback so that isodifications can be ma^e. 

This information is also used to satisfy the funding agency's expectation 
for a formal evaluation. At times this information is used to inflcancy key 
decision-makers when making a go-*no--go decision. Howeveri this is an indirect 
function of the independent program evaluator. 

The independence of the independent program evaluator maximizes the 
objectivity of the findings in both process and product evaluation. Howeveri 
quite often this Is not the key factor that actually influenced the operation 
of the project. In this case ve sacrifice usefulness and credibility of the 
Infom^tiou to maintain our objectivity and independence. 

13 
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The benefits derived from Independent program evaluators can be improved 
by developing sotmd working relationships with the key people of the various 
projects* Some techniques used to do this are more site visits^ more Interim 
cooounlcatlon and nonthly or bl«*monthly meetings. 

In summary, the Independent project evaluator's role has been defined 
to stress objectivity and independence* At times , these attributes have 
not contributed directly to cnanges in project behavior. This is largely 
due to the lack of relationships with project staff. There has been on attempt 
in the Program Evaluation Branch of the Cincinnati Public Schools to maintain 
the objectivity and independence characteristics and at the same time build 
better relationships with project staff. 

External Evaluator . The external evaluator provides information on the 
generalizabllicy of federally fimded projects in the Cincinnati Public Schools. 
The purpose of the external evaluator is to provide insight into the Impact 
of a program and to provide key declsion*-makers with information to determine 
the future of a project. Rather than responding to the specific objectives 
of the project 9 this evaluator responds to the impact questions of key decision-* 
makers* This evaluator 's function is not directed toward program improvement 
but rather toward answering questions related to Che impact of t a project 
on students » school » school system and community. This type of evaluation 
is intended to occur during the final year of a project. During the process , 
the evaluator has little or no contact with field staff. 

The continuation of a project is dependent on this critical stage, since 
major expenditures cf hard money could be an outcome ot this evaluation. The 
data rust be bbjectlven The external evaluator's role is highly independent, 
since the external iwaluator has little or no contact with project personnel. 
The usefulness of this report is limited to the V^y decision-makers making 
the go-*no-*go declsloii. 
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Although this Is a concept that has been used in Cincinnati Public Schools ^ 
project funds are usually not available for this type of function* Secondly^ 
key decision-makers are relying more heavily on evaluation reports of 
indepeudent program and project evaluatore* 

Educational Accomplishment Auditor ^ An educational accomplishment auditor 
was utilized on those Title III^ section 306 projects which included funds for 
an external audit* The audit was contracted with a regional university* The 
auditor and project manager attended the United States Office of Education 
training session which outlined the role of the auditor* The purpose of an 
educational accomplishment audit is to provide an objective and independent 
verfiLcation of a project's evaluation process, to assess the appropriateness 
of the projects procedure for evaluating product , to assess the appropriateness 
of management procedures, and to verify the accuracy of the projects evaluation 
report « 

The auditor is typically not involved in the identification of the 
problem or the treatment* Ue/she conducts a pra^-audit of the project evaluation 
plan* The pre-*audit can contain recommendations for strengthening the design, 
improving instruments, or improving data collection* During a project year, 
the auditor conducts one or two site visits to assess the implementation of 
the evaluation plan* During these visits the auditor interviews the project 
coordinator, evaluator, staff members, and other individuals in the setting 
served by the v^oJ^ct* 

The auditor makes recommendations on project management and evaluation, 
particularly in those areas covered by specific "program management objectives"* 
The greater part of the auditor's time is spent on validating the evaluation 
of the outcome objectives of the project* The auditor identifies problem 
areas but makes no suggestions on program modifications* In some cases the 
auditor has provided a reaction to program modification prepared by the project 

rn^/-> coordinator* ^ ^ 
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The final audit report is prepared as a supplement to the annual 
evaluation report* The auditor discusses his findings with the project 
staff after the document has been completed« 

The auditor's role is defined so that his objectivity and independence 
are maximized* The auditor is constrained » however » to deal with the 
objectives of the project » and the evaluation of those objectives* In 
cases where the objectives of the project are inadequate^ the auditor's 
impact on the project is weakened* The relatively small amount of "on-site" 
experience reduces the credibility and potential usefulness of his recom- 
mendations in the area of project management* To the extent that the problems 
are widely accepted by the staff , as problems ^ the re ^commendations can be 
useful* 

Summary 

The Department of Research and Development of the Cincinnati Public 
Schools utilizes five evaluation roles; project evaluator^ local school 
evaluator^ independent program evaluator, external evaluator^ and external 
auditor* Each evaluation role would like to produce objective^ indepedent^ 
credible^ and useful evaluations* However, in practice, all of the above 
characteristics cannot be maximized at any one time* The safeguards which 
insure credibility and independence are much concerned with the relationship 
between the evaluator And the user* The role definition of the evaluator 
vii:hin the organization also greatly affects these two characteristics* On 
the other hand, the safeguards which insure objectivity and usefulness are 
concerned with the quality of the data provided and the responsiveness of the 
data to the particular uscir'c needs* 
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